Papers with computation per token
Adaptive Gating in Mixture-of-Experts based Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models employ a fixed gating network where each token is computed by the same number of experts. |
| Approach: | They propose a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution. |
| Outcome: | The proposed model reduces training time and inference quality while maintaining sparsity while maintaining inference accuracy. |